ENH: add remfile backend with pluggable backend architecture - #131

Bug: the HEAD request breaks fstat() on open FUSE handles when the server rejects HEAD.

This is not "rarely called". DataLadFUSE.getattr(path, fh) takes the "File already open" branch and calls file_getattr(f) → f.info() whenever the kernel asks for attributes of an open handle (fgetattr). I saw this in a real mount log while h5py was opening ros3test.nwb. It does not depend on the key carrying a size.

The HEAD is also the request remfile itself avoids, because presigned S3 GET URLs (e.g. what DANDI's api.dandiarchive.org/.../download/ redirects to) reject HEAD. When that happens, urllib raises HTTPError, which is not a FileNotFoundError, so file_getattr does not map it to ENOENT and fusepy gets an unhandled exception.

Reproduced deterministically: a dataset whose only URL is a local server that answers HEAD with 403 and serves ranged GETs. I called ops = DataLadFUSE(ds, caching=False, backends=...), then fh = ops.open(path, O_RDONLY), then ops.getattr(path, fh):

== remfile,fsspec
open -> 1000 RemfileWrapper
getattr(path, fh) FAILED: HTTPError HTTP Error 403: Forbidden
== fsspec
open -> 1000 HTTPFile
getattr(path, fh) size: 177728

fsspec survives because HTTPFile.info() returns the details it cached at open time. remfile also already knows the size, since RemFile.__init__ fetched Content-Length with an aborted GET. So no network call is needed here:

With this change, the urllib.request import and the two test_info_* tests that mock urlopen need adjusting.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. info() now returns {"type": "file", "size": self._f.length} and makes no request at all. The urllib.request import stays in the module, but only for the new size probe in open_url() (see the reply on the retry/stall thread). The two test_info_* tests that mocked urlopen are replaced by test_info_uses_remfile_length, which asserts urlopen is not called.

Bug: read() with no size returns b"" and moves the position backwards, and reading at EOF can raise.

RemFile.read() does not support size=-1. With -1, its chunk range comes out empty, so it returns b"" and then does self._position += size. Against the real S3 object:

f.seek(10); f.read()          -> 0 bytes; tell() == 9
fsspec: g.seek(10); g.read()  -> 177718 bytes

RemoteFilesystemAdapter.open() returns this object typed as IO, so any API user calling .read() silently gets empty data.

RemFile.read(n) has two related problems:

• It always advances by n, even past EOF. read(100) at length-5 leaves tell() == length+95.
• When the size is an exact multiple of its 100 KiB chunk, a read at EOF issues an unsatisfiable Range request instead of returning b"". That breaks __next__'s EOF detection, so fsspec-head -n fails. On a 204800-byte .h5 served locally:

iteration FAILED at EOF: _PermanentHTTPError Error fetching bytes 204800-204799: 416 Range Not Satisfiable

Clamping to the remaining length fixes all of these:

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, with the suggested clamp. __next__ now reads through self.read() rather than self._f.read(), so line iteration gets the clamp too — otherwise the 416 at EOF still broke fsspec-head -n.

The mock in _make_mock_remfile was made faithful to remfile.RemFile rather than well-behaved (requires an explicit non-negative size, advances past EOF, raises 416 at EOF, seek() returns None); with that, the existing test_iteration/test_iteration_empty went red before the fix. Checked against the real S3 object too: seek(size); read(10) → b"", over-long read returns the tail and leaves tell() at EOF, read() with no size returns all 4008 bytes.

Minor: RemFile.seek() has no return statement, so this returns None despite -> int. mypy doesn't catch it because self._f is Any. In the test drive, f.seek(100) gave None, while io.BytesIO(...).seek(100) gives 100.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, as suggested. Covered by test_seek_returns_new_position.

Unreachable URLs stall about 30× longer than with fsspec, and in FUSE that blocks the whole mount.

remfile.File(url) probes Content-Length and retries transient failures (connection errors, 5xx) 8 times with exponential backoff, about 25 s plus connect time. Only 4xx fails fast. The adapter tries URLs one after another, so each dead URL costs that much before the next one is tried.

Test: dandiset 000027. Its first whereis URL is api.dandiarchive.org/..., which my sandbox's network policy blocks. Its second URL is a versioned dandiarchive.s3 one. Running datalad fsspec-head -c 8 sub-RAT123/sub-RAT123.nwb:

In fusefs, DataLadFUSE.open() calls self._adapter.open() while holding the global rwlock, so every other read on the mount waits the whole time. Dead mirrors, firewalled hosts, offline special-remote URLs and the several candidate URLs that get_urls() builds per HTTP remote all hit this.

Possible fix: do a cheap probe here with no retries and a short timeout, for example a ranged GET bytes=0-0 that also reads Content-Range to get the size, so URL fall-through stays fast. Then pass the known size to remfile (key.size is almost always available) so it skips its own probe. remfile only exposes that as the underscore-private _size=, though. It may be worth asking remfile upstream for public size= and retry parameters, or pinning the remfile version.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching, along the lines you suggested. RemfileBackend.open_url() now does the cheap probe itself — one GET bytes=0-0 with a 10 s timeout and no retries — reads the total from Content-Range (falling back to Content-Length when the server ignores Range, and to None when it says neither), and passes it to remfile.File(..., _size=...) so remfile skips its own retrying probe. For a live URL that is the same single round trip it was already paying.

Measured against a missing object in the same bucket: 0.20 s, down from ~30 s. The pinned-S3 integration tests still pass unchanged.

_size= is private API, so setup.cfg now pins remfile ~= 0.1.15. Asking upstream for a public size= (and retry parameters) still seems worth doing — happy to file that separately.

Dispatching only on key.suffix misses HDF5 files whose keys come from a non-*E backend. git annex addurl --fast/--relaxed (and datalad addurls --fast) produce URL/VURL keys, and AnnexKey.parse sets suffix=None for those:

fast.nwb      VURL-s177728--https://dandiarchive.s3.amazonaws.com/ros3test.nwb
relaxed.hdf5  VURL--https://dandiarchive.s3.amazonaws.com/ros3test.hdf5
fast.nwb: backend remfile cannot handle (suffix=None, mode=rb)
fast.nwb: opening via backend fsspec

The dandisets/* datasets use SHA256E, so they are fine. DatasetAdapter.open() already has relpath, though. Passing it to can_handle() (e.g. can_handle(key, mode, relpath)) and falling back to PurePosixPath(relpath).suffix when the key has no suffix would make the choice follow what the user sees in the tree.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. Backend.can_handle() takes a third relpath argument (optional, so the signature stays usable standalone), and RemfileBackend falls back to PurePosixPath(relpath).suffix when the key carries no suffix. DatasetAdapter.open() passes the relpath it already has. A key suffix still wins when present.

--caching=ondisk is now silently a no-op for HDF5 files once remfile is installed. caching is passed only to FsspecBackend. With the new default remfile,fsspec, .nwb/.h5 files bypass the on-disk cache entirely:

datalad fsspec-head --caching ondisk --backends remfile,fsspec -c 8 ros3test.nwb  -> 0 files in .git/datalad/cache
datalad fsspec-head --caching ondisk --backends fsspec         -c 8 ros3test.nwb  -> 2 files in .git/datalad/cache

So installing datalad-fuse[full] quietly changes what --caching ondisk does for exactly the large files where it matters most. Options:

• wire remfile.DiskCache into e.g. .git/datalad/cache/remfile, and clear it in RemfileBackend.clear() so fsspec-cache-clear and datalad.fusefs.cache-clear keep working;
• have RemfileBackend decline when caching=True;
• at least document it, and update the --caching help, which still says "cache fsspec'ed files".

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching — the first of your three options. RemfileBackend now takes (path, caching) like FsspecBackend, builds a remfile.DiskCache at .git/datalad/cache/remfile when caching is on, passes it to remfile.File(..., disk_cache=...), and removes the directory in clear(), so fsspec-cache-clear and datalad.fusefs.cache-clear cover HDF5 files again. Verified end to end against the real S3 object: a 100-byte read writes a chunk under that directory, and clear() removes it.

The --caching help text still says "fsspeced" — that and the README are in the follow-up noted on the PR.

The global datalad.cfg doesn't read the dataset's own .git/config or .datalad/config. So git config datalad.fusefs.backends fsspec inside a dataset is ignored, whether the command runs from inside it or with -d. I checked both: remfile was still used. This matches how datalad.fusefs.cache-clear is read today, so it is not a regression. DatasetAdapter.__init__ already builds Dataset(path), though, so ds.config.get("datalad.fusefs.backends") would honor per-(sub)dataset settings at no extra cost.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. resolve_backends() takes an optional config, and DatasetAdapter.__init__ passes ds.config. That is a superset of the global one rather than a replacement: ConfigManager.__init__ does self.overrides = datalad.cfg.overrides.copy(), so -c datalad.fusefs.backends=... still wins. The global cfg remains the fallback when no config is supplied.

I left datalad.fusefs.cache-clear alone, since that is outside this PR.

Minor (efficiency): when the first backend exhausts its URLs, the next backend calls get_urls() and get_exporttree_urls() again. That repeats the git annex whereis subprocess (batch=False), the two examinekey calls and, on the exporttree path, the boto3 list_object_versions call. Collecting the URLs lazily into a list the first time and replaying it for later backends would avoid that, and the lazy boto3 behaviour would stay the same.

Generated by Claude Code

Fixed in d7fb6b6 BF: address review of PR #131 — RemfileWrapper semantics, URL probing, caching. Added a small replayable() helper: the chained get_urls() + get_exporttree_urls() iterator is pulled at most once, and each backend replays what has already been seen before pulling anything new — so the laziness of the boto3 call is unchanged, but whereis/examinekey/list_object_versions run once per open() rather than once per backend. test_urls_enumerated_once_across_backends pins the call count; test_urls_not_enumerated_when_no_backend_handles pins the laziness.

--caching ondisk can now serve corrupted data. remfile.DiskCache writes each entry in place and get() returns whatever is on disk; remfile uses it without checking the length. So an entry cut short by an interrupted write is later served as file data. A killed fusefs process or a full disk can produce such an entry.

Verified on b4dbee6. After truncating one entry under .git/datalad/cache/remfile to half its size, datalad fsspec-head --caching ondisk -c 177728 ros3test.nwb returns 140064 bytes with a different checksum, and no error.

The proper fix belongs in remfile and will be reported there. In the meantime, the size that _probe_size() returns lets the backend reject damaged entries. remfile then fetches the chunk again and its set() rewrites the entry:

class _CheckedDiskCache:
    """Never return a remfile.DiskCache entry of the wrong length."""

    def __init__(self, cache: Any, size: int) -> None:
        self._cache = cache
        self._size = size

    def get(self, key: str) -> Optional[bytes]:
        value = self._cache.get(key)
        if value is not None:
            # remfile's key is "<url>|<chunk size>|<chunk index>"
            chunk_size, index = (int(x) for x in key.rsplit("|", 2)[1:])
            if len(value) != min(chunk_size, self._size - index * chunk_size):
                return None  # damaged (e.g. interrupted write): fetch again
        return value

    def set(self, key: str, value: bytes) -> None:
        self._cache.set(key, value)

        size = self._probe_size(url)
        disk_cache = self._disk_cache
        if disk_cache is not None and size is not None:
            disk_cache = _CheckedDiskCache(disk_cache, size)
        # `_size` is private API; see the remfile pin in setup.cfg.
        remfile_obj = self._remfile.File(url, _size=size, disk_cache=disk_cache)

I checked this directly against remfile 0.1.15 and the real S3 object, with both chunk entries truncated:

• plain DiskCache: wrong data, 88864 bytes;
• with the wrapper: correct data, 177728 bytes, and the entries were rewritten to 102400 and 75328 bytes;
• a third read was also correct.

Like the _size= call, this relies on remfile's 0.1.x key format, which the remfile ~= 0.1.15 pin covers.

Generated by Claude Code